Message communication techniques

ABSTRACT

A network protocol unit interface is described that uses a message engine to transfer contents of received network protocol units in message segments to a destination message engine. The network protocol unit interface uses a message engine to receive messages whose content is to be transmitted in network protocol units. A message engine transmits message segments to a destination message engine without the message engine transmitter and receiver sharing memory space. In addition, the transmitter message engine can transmit message segments to a receiver message engine by use of a virtual address associated with the receiver message and a queue identifier, as opposed to a memory address.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a Continuation of U.S. patent application Ser. No.13/173,459, filed Jun. 30, 2011, which is a Continuation of U.S. patentapplication Ser. No. 12/319,099, filed Dec. 30, 2008, and entitled“Message Communication Techniques,” which is related to co-pending U.S.patent application Ser. No. 12/319,033 filed Dec. 30, 2008, entitled“Interrupt Techniques,” inventors Kumar et al.

FIELD

The subject matter disclosed herein relates generally to techniques fortransmitting data.

RELATED ART

In data networking, some data transfers occur by identifying a memorylocation of data and allowing a recipient to access the data from thememory location. One example is a soft-switch virtual machine deliveringEthernet packets to guest virtual machines (VMs) in a virtualizedplatform. The soft switch and guest VM can use page flipping, doublecopy through a shared staging buffer, or a hypervisor copy to transferthe packet. Another example is the iWARP specification described in JeffHilland, RDMA protocol verbs specification (version 1.0) (2003).

Each of these alternatives carries high processing cost out ofproportion to the simple goal of moving data. In addition, sharingmemory space can create issues. When a memory space is corrupted, eachsoftware or hardware that accesses the memory space may malfunction. Inaddition, as the number of cores in a central processing unit (CPU)grows, the likelihood that efficient intervening memory existsdecreases. For example, a shared cache between sender and receiver maynot exist, forcing interaction in DRAM.

As another example, consider a traditional data copy of a buffer from asender to a receiver. If the sender performs the copy, the destinationbuffer becomes pure cache pollution in the sender's data caches. If thereceiver copies, the source buffer becomes pure cache pollution in thereceiver's data caches. Such misuse of cache is difficult or impossibleto eliminate in today's CPU architectures. In some cases, it isdesirable to permit transfer of data without sharing memory space.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the present invention are illustrated by way of example,and not by way of limitation, in the drawings and in which likereference numerals refer to similar elements.

FIG. 1 depicts a system in accordance with an embodiment.

FIG. 2 depicts an example of communications between message engines inaccordance with an embodiment.

FIG. 3 depicts an example of initialization process for transmissions ofmessage segments from a VMTE to a VMRE in accordance with an embodiment.

FIG. 4A depicts a simplified block diagram of a message engine inaccordance with an embodiment.

FIG. 4B depicts in block diagram format a network communication systemthat utilizes message engines to communicate with external devices inaccordance with an embodiment.

FIG. 5 depicts a high level block diagram of a message engine inaccordance with an embodiment.

FIG. 6 depicts an example format for a context to define an availableVMRE in accordance with an embodiment.

FIG. 7 depicts an example format for a no-operation command in a sendqueue in accordance with an embodiment.

FIG. 8 depicts an example format for a command in a send queue inaccordance with an embodiment.

FIG. 9 depicts an example receive queue format in accordance with anembodiment.

FIG. 10 depicts an example message segment format in accordance with anembodiment.

FIG. 11 depicts an example request to send (RTS) message format inaccordance with an embodiment.

FIG. 12 depicts an example clear to send (CTS) message format inaccordance with an embodiment.

FIG. 13A depicts an example flow diagram of a process to transfermessages from an IO device using a message engine in accordance with anembodiment.

FIG. 13B depicts an example flow diagrams of a process to receivemessages using a message engine at an IO device in accordance with anembodiment.

DETAILED DESCRIPTION

Reference throughout this specification to “one embodiment” or “anembodiment” means that a particular feature, structure, orcharacteristic described in connection with the embodiment is includedin at least one embodiment of the present invention. Thus, theappearances of the phrase “in one embodiment” or “an embodiment” invarious places throughout this specification are not necessarily allreferring to the same embodiment. Furthermore, the particular features,structures, or characteristics may be combined in one or moreembodiments.

FIG. 1 depicts a high level overview of a system that uses MessageEngines (ME) in accordance with an embodiment. Other more detailedaspects of message engine capabilities are described with regard toother figures. In this example, a first message engine, ME1, is capableof transmitting messages to a second message engine ME2 using aninterconnect and without ME1 and ME2 using the same memory space. Insome embodiments, a “message” encapsulates any type of data, but may notcontain memory addresses for source memory buffers of the data and maynot identify destination memory addresses for the data. In oneembodiment, ME1 may have knowledge of a virtual message engine address(VMEA) of ME2 but not know the destination memory address of a memorybuffer used by ME2 to store data transmitted by ME1. In addition, ME2may know the VMEA of ME1, but not know a memory address from which datais transmitted using ME1. As will be described later, a VMEA mayidentify a particular message engine. A VMEA may be distinct from amemory address.

A benefit of not sharing memory space is that memory isolation fordifferent applications can be maintained. A benefit of memory isolationis that failure or corruption of a memory space only affects theapplication that uses that memory space and no other applications.Another benefit of memory isolation is elimination of hardware andsoftware overhead used to permanently or temporarily create sharedmemory between the memory spaces, or to transfer ownership of pages ofmemory between the memory spaces, or to transition to higher softwareprivilege levels needed to copy data directly between the memory spaces.

In an embodiment, message engines may not maintain coherency of memory.Coherency is a protocol that ensures that multiple accessors of memoryread the same data using the same address. Accordingly, by notmaintaining coherency of memory, different message engines do not incurthe overhead of the coherency protocol. Individual message engines mayoperate in distinct coherency domains. Coherency may be maintained ateach endpoint. For example, coherency may be maintained between a threadand a message engine.

In an embodiment, message engines do not share the same physical memory.For example, message engines may not be permitted to access the same RAMchip.

A Virtual Message Engine (VME) is an instance of a virtualized interfaceto a Host Physical Message Engine (HPME). A VME supports sending andreceiving of messages from virtual memory. Message Engines may alsosupport RDMA Write and RDMA Read operations. A VME is either a VirtualMessage Transmit Engine (VMTE) or Virtual Message Receive Engine (VMRE).A hypervisor or other privileged system entity (e.g., kernels in anative OS) may schedule one VMTE at a time, but multiple VMREssimultaneously on a single HPME. The hypervisor or other privilegedentity may be a software routine executed by a processor.

Two virtual machines may make forward progress in their communication solong as a VMRE is able to receive when a VMTE transmits. The hypervisormay schedule the VMTE in a fine grain manner, such as by co-schedulingit with an associated software thread. The VMTE may discontinuetransmitting when not scheduled by the hypervisor. The HPME may supportmultiple VMREs simultaneously. The VMRE may be scheduled independentlyfrom the CPU threads associated with a VMTE or a VMRE. The persistenceof the VMRE may help with forward progress across scheduling quanta.

In some embodiments, a VMRE or VMTE may use processor virtual addressesfor receiving and sending messages. These virtual addresses may use thesame memory translation mechanism as software threads. The use ofvirtual addresses for messaging assists with address space isolationwithout the overhead of a separate memory control mechanism redundant tothe existing software memory translation mechanism.

The hypervisor may build descriptors for VMEs (e.g., VMREs and VMTEs)with virtual addresses native to the address space in which the VMElogically exists. A VME may logically exist in any virtual addressspace, for example: kernel mode of a native operating system (OS),kernel mode of a para-virtual non-VT guest OS, kernel mode of a fullyvirtual guest OS, user mode of a native OS, user mode of a para-virtualnon-VT guest OS, user mode of a fully virtual guest OS, or a hypervisor.Virtual Message Engine Addresses (VMEA) may identify the VMTE or VMREassociated with a message.

In the example of FIG. 1, ME1 represents a Host Physical Message Engine(HPME). ME1 may be physically and logically associated with a source orsink of data, such as a computational element or input/output (IO orI/O) device (e.g., a network interface). ME1 may be incorporated into anIO device. A Virtual Message Transmit Engine (VMTE) associated with ME1may use the VMEA of the destination VMRE to send messages to thedestination VMRE. Accordingly, on transmit, ME1 maps a VMEA to anappropriate link over which segments flow and the segments arrive at thecorrect VMRE.

In this example, ME2 also represents an HPME. To receive segments, theME2 uses a VMEA to locate the VMRE of the incoming message segments fromthe set of all VMREs scheduled at ME2. ME2 is capable of receivingmessages for processing by multiple hardware threads, HT0 and HT1. ME2may be incorporated into a core with one or more hyperthreads. In thisexample, hyperthreads HT0 and HT1 are two hyperthreads that can processreceived messages. ME2 may be located outside the core or may beassociated with more than one core. Message engines may exist outside ofthe CPU socket, for example in discrete IO devices, so long as theintervening physical and link-layer interconnect can transport messagesappropriately.

ME2 may place received messages into various levels of the cachehierarchy or memory. For example, received messages can be stored in anyof a data cache unit (DCU), mid level cache (MLC), last level cache(LLC) shared by all cores, or main memory (e.g., DRAM or SRAM). The DCUmay be the fastest cache nearest to a software pipeline.

ME2 may be virtualized. As with a software thread, a Message Engine fora core may interact with system software via virtual addresses asdefined by traditional memory management unit (MMU) page tables. Messageengines may share virtual address space with one or more threads, thoughsystem software could construct a dedicated set of page tables for amessage engine.

A Message Engine may subdivide a message into one or more messagesegments suitable for transport over a message interconnect. Messagesegments may have no associated memory address, may be un-ordered withrespect to memory transactions, and travel out-of-band with respect tothe cache hierarchy. An application or other software that is to processthe received message segments may specify destination memory addressesin which the receiving message engine may store the message segments.

The interconnect may provide a transport medium for message segmentsfrom a sending message engine to a receiving message engine. Theinterconnect may share physical resources with a coherent memoryinterconnect, but provides a transport that is logically distinct andout-of-band with respect to coherent memory. Example interconnectsinclude a ring, crossbar, and/or mesh. Interconnects may also includeexternal buses such as PCI express.

Although not depicted, the system of FIG. 1 may also include thecapability to access a storage device using a storage adapter. Forexample, the storage adapter may be capable of communicating with thestorage in conformance with any of the following protocols: SmallComputer Systems Interface (SCSI), Fibre Channel (FC), and/or SerialAdvanced Technology Attachment (S-ATA). The storage may be implementedas a non-volatile storage device such as but not limited to a magneticdisk drive, optical disk drive, tape drive, an internal storage device,an attached storage device, flash memory, battery backed-up SDRAM(synchronous DRAM), and/or a network accessible storage device.

FIG. 2 depicts an example of communications between message engines inaccordance with an embodiment. A virtual message transmitter engine(VMTE-A) is associated with a host physical message engine HPME-A. Ahypervisor (not depicted) is responsible for one-time establishment of aconnection between sending and receiving message queues. Thereafter,unprivileged software in the address space of the VME may populate thequeues with buffers. Once software creates a Send Queue Entry in a SendQueue (SQ), the VMTE may begin the transmit process described in anembodiment in FIG. 2.

At 201, VMTE-A initiates a transfer of message segments to a receivermessage engine by transmitting a request to send (RTS) message to theVMRE. An RTS message may have the format described with regard to FIG.11. In this example, the physical receiving message engine is HPME-B. Avirtual receive message engine VMRE-B is associated with HPME-B.

HPME-A may use the following process to transmit an RTS message.

1. Allocate storage for a returning clear to send (CTS) message. Thisstorage may be a portion of the memory occupied by the SQ entry for thismessage. The transmitter may associate the request ID (RID) value withthis CTS storage in order to correctly process a returning CTS message.

2. Initialize the RTS message with the Destination and Source VMEAddresses, queue number (QN), message sequence number (MSN), and requestID (RID).

3. Transmit the RTS message.

At 202, HPME-B may perform the following checks on the receivedmessages: 1) the Destination VME Address belongs to a VMRE scheduled onthe PME; 2) the QN exists and is less than or equal to the maximumpermissible QN for the VMRE; 3) the Source VME Address is permitted tosend to the specified QN at the destination VMRE address; 4) the MSN isgreater than or equal to the minimum MSN value for the QN; and 5) theMSN is less than or equal the maximum MSN value for the QN. If allvalidation checks succeed, then HPME-B requests a reassembly slot usedto reassemble received message segments.

The process to allocate a reassembly slot may be as follows.

1. Provide the Source VME Address, QN and MSN, RID values to aReassembly Reservation Station (not depicted).

2. The Reassembly Reservation Station attempts to allocate a reassemblyslot. If a slot is available, the Reassembly Reservation Stationprovides the Reassembly Tag and the RID to the CTS Transmit Handler (notdepicted). The Reassembly Reservation Station may store pertinent localinformation such as the Source VME Address, QN and MSN values in acontext associated with the Reassembly Tag.

3. A CTS Transmit Handler prepares the Segmentation and Reassembly Layerto handle inbound data segments associated with the RT.

4. The CTS Transmit Handler constructs and transmits the CTS message.The RID field contains the verbatim value received from the sender inthe associated RTS message.

A Segmentation and Reassembly Layer (not depicted) of the sender of aCTS message may be ready to accept incoming data segments for theassociated Reassembly Tag immediately upon transmitting the CTS message.

At 203, VMRE-B permits transmission of messages from VMTE-A byinitiating transmission of a clear to send (CTS) message through avirtual message transmitter engine (not depicted) associated withHPME-B. The CTS message includes the Reassembly Tag (RT) value used bythe VMRE to recognize this message from other in-flight messages.

VMRE-A (not depicted) is associated with HPME-A and is used to processreceived messages. At 204, VMRE-A (not depicted) recognizes the CTSmessage from HPME-B. The VMRE-A may use the RID value in the CTS messageto identify the associated message.

At 205, VMRE-A marks the message as ready-to-send. If VMTE-A iscurrently scheduled, then VMTE-A begins transmission of the messagedepicted here as containing two segments called TX and TX-L from memoryregion A (memory-A) to memory region B (memory-B) using a data mover. Adata receiver such as unprivileged software (not depicted), that is toprocess contents of the messages, populates a receive queue (RQ) withdescriptors that point to buffers in memory. A reassembly layer readsthe descriptors, the Reassembly Tag (RT) and Message Segment Offset(MSO) from message segments and informs a data mover to place thesegments in a buffer designated memory-B. The descriptors, RT, and MSOidentify a destination for segments as memory-B. The reassembly layermay inform VMRE-B when all segments of a message have been placed inmemory.

VMTE-A transmits segments observing transmit priority with respect toother in-flight message transmit operations. If VMTE-A is not currentlyscheduled when VMRE-A marks the message ready-to-send, then transmissionof the message may resume after the hypervisor reschedules VMTE-A.

At 206, message segments, TX and TX-L, arrive at memory region B. Themessage segments include the RT field used by VMRE-B to identify themessage context to which the segments belong. This context may includethe source and destination virtual message engine addresses, queuenumber (QN) field, and the message sequence number field (MSN).

The recipient PME locates the VMRE associated with the RT. If the PMEcannot locate the receive context, the PME discards the segment. Therecipient PME also validates that the MSO specified in the segment iswithin range for the message being received. If the MSO is not in range,the PME discards the segment.

The VMTE may transmit segments in order, but reordering of segmentswithin a message may occur. At the message destination, a ReassemblyLayer (not depicted) may: 1) place segments in memory; 2) may indicatemessage arrival to the VMRE only when all of the following conditionsare met: a) all segments of a message have been placed in memory and b)all segments of all previous messages have been placed in memory; and 3)when indicating message arrival, the Reassembly Layer may indicate tothe VMRE whether a message is a control messages or a data message.

The recipient of a message segment may ensure the segment is placedwithin the correct location in the message, regardless of arrival order.An MSO field in the message makes placement of a segment in the correctlocation a convenient operation in the VMRE. However, the recipient mayensure that all segments of a message have arrived before indicatingthat the message is available for processing.

The VMRE interface may make the following assertions.

1) After software posts a receive queue entry (RQE), but before a VMREindicates a reconstructed message has arrived and is available forprocessing, the VMRE may manipulate memory within a message data bufferin any manner. Software may not depend on any particular data accessingbehavior in the data buffer.

2) VMRE may manipulate memory within the RQE in any arbitrary manner.Software may not depend on any particular data accessing behavior in theRQE.

3) The VMRE may manipulate memory within the RQ Header to increment thehead field by 1 or more.

4) After a VMRE indicates a message has arrived, software (e.g.,software that processes the received message) can assume that allcompleted messages are placed in memory as indicated by the Head fieldof the RQ Header.

FIG. 3 depicts an example initialization process for transmissions ofmessage segments from a VMTE to a VMRE in accordance with an embodiment.At 301, a kernel/operating system (OS) requests a hypervisor for aconnection using message engines.

At 302, the hypervisor builds a table with contexts that describeavailable VMREs. Contexts are accessible by message engines. In someembodiments, suitable contexts are those described with regard to FIG.6.

At 303, the hypervisor transfers to the kernel the logical addresses ofthe virtual message receive engine (VMRE) and virtual message transmitengine (VMTE) of a connection. The VMRE and VMTE correspond to addressesthat serve as logical interfaces to physical message engines involved inthe message engine connection.

At 304, the kernel requests the hypervisor to connect its send queue(SQ) to a remote receive queue (RQ).

At 305, a message receiver that controls receipt of messages indicatesreadiness to receive messages. The message receiver could be, but is notlimited to, application software, kernel, soft-switch, or a fixedfunction accelerator.

At 306, the hypervisor allocates an SQ, RQ, and completion queue (CQ)and indicates the allocated SQ, RQ, and CQ to the kernel.

Thereafter, a physical transmitter message engine transfers contents ofidentified in part using the SQ to a location identified in part usingthe RQ. The RQ may exist in pageable memory. In one embodiment, thetransmitting message engine uses a data mover to form message segmentsfrom contents of linear addresses, where the linear addresses areidentified by the SQ. The transmitting message engine uses the datamover to place message segments into internal buffering pendingtransmission on the message interconnect. The receiving message engineuses a data mover to place message segments into linear addressesidentified by the RQ. Linear addresses are contiguous addresses that aremapped in a CPU's page tables and these linear addresses may nativelyaccessible by software.

A Completion Queue (CQ) allows a VME to notify software of activity,such as receiving or transmitting a message. A CQ may be associated withone or more SQs or one or more RQs. A CQ and its associated RQs or SQsmay exist in the same address space as the VME. A CQ may reside incontiguous virtual address space. A CQ may exist in pageable memory andthe VME may incur a page fault attempting to read or write the CQ.

Page faults may occur when a VME accesses the virtual buffers used tosend and receive messages for a particular queue. A VMTE with a messageto send will begin segmenting the message and providing those segmentsto the PME. The PME will transmit segments to the recipient throttled bylink credit and in observation of transmit queue prioritization.

If the VMTE encounters a page fault while reading the message buffer,the VMTE takes the following actions: 1) pushes its current context intoa Fault Context Buffer (not depicted) for this SQ; 2) halts transmissionof messages from the faulting send queue; 3) interrupts the threadassociated with the VME to resolve the fault; and 4) resumes processingall other Send Queues as normal.

For its part, the fault handler thread may take the following actions:

1. Read the fault context buffer and load the faulting page into memory.

2. Write the VME doorbell of the faulting SQ to resume messagetransmission.

Upon detecting the doorbell, the VME may take the following actions:

1. Continue processing higher priority SQs as normal.

2. After no higher priority SQs have a message to send, the VMTE loadsthe fault context buffer for the faulting SQ.

3. Resume segmenting the message starting with the first faultingsegment.

A VMRE reassembles segments received from the PME by writing thosesegments to the corresponding message buffer in memory. If the VMREencounters a page fault while writing a Receive Queue, the VMRE may takethe following actions.

1. Push its current context into the Fault Context Buffer for this RQ.

2. Transmit the FAULT message to the sending VMTE. The FAULT messageindicates the MSN of the faulting message segment.

3. Interrupt the thread associated with the VMRE to resolve the fault.

4. Discard any further message segments received for this message.

5. Continue to accept and place message segments for other queues(non-page faulting) as normal.

A fault-handler thread may take the following actions:

1. Read the Fault Context Buffer and faulting page into memory.

2. Construct a message to the sender to inform the sender to resumetransmission of the faulting message. The contents of this messagedepend on the specific nature of the VMTE.

FIG. 4A depicts a simplified block diagram of a message engine inaccordance with an embodiment. For example, message engine 400 can beused to transmit messages to any other message engine, such as a may becontained within a network interface. Message engine 400 can also beused to receive messages from another message engine. The networkinterface may be capable of transmitting and receiving network protocolunits. As used herein, a “network protocol unit” may include any packetor frame or other format of information with a header and payloadportions formed in accordance with any protocol specification.

I/O interface 402 may perform media access control (MAC), filtering, andcyclic redundancy check (CRC) operations on received Ethernet frames aswell as media access control for Ethernet frames to be transmitted. Inother embodiments, I/O interface 402 may perform protocol encoding anddecoding for frames and packets of other specifications.

Buffer 403-A may store received Ethernet frames processed by I/Ointerface 402 whereas buffer 403-B may store Ethernet frames that are tobe transmitted prior to processing by I/O interface 402.

Message segmentation block 404 is capable of segmenting Ethernet framesfrom buffer 403-A into messages of a size compatible with an underlyingmessage interconnect. Message segmentation block 404 may query messageroute table 410 to determine a Virtual Message Receive Engine (VMRE),queue number (QN), and message sequence number (MSN) in which totransfer messages that transport contents of a received Ethernet frame.Message segmentation block 404 may transfer message segments that are tobe transmitted into buffer 407-A. Buffer 407-A may be identified using asend queue (SQ). Message segmentation block 404 may transfer messagesegments from buffer 407-A to the location associated with a VMRE, QN,and MSN using a data mover (not depicted).

Buffer 407-B may store message segments received through aninterconnect. Buffer 407-B may be identified in part using a receivequeue (RQ). Message reassembly block 406 may transfer message segmentsto buffer 407-B using a data mover (not depicted).

Message reassembly block 406 is capable of reassembling message segmentsin buffer 407-B into complete messages and providing the contents inbuffer 403-B for transmission in one or more network protocol units.

Interface 408 may transfer messages from message segmentation block 404to an interconnect and transfer messages from an interconnect to messagereassembly block 406.

FIG. 4B depicts in block diagram format a network communication systemthat utilizes message engines to communicate with external devices inaccordance with an embodiment. For example, system 450 may include thecapability of an Ethernet compatible network interface that transfersreceived Ethernet frames using receive (RX) message engine 454. System450 also uses a transmit (TX) message engine 456 to receive messagesthat contain data to be transmitted in Ethernet frames or control ormanagement information for the Ethernet network interface. System 450may encode and decode other types of network protocol units such as butnot limited to Serial ATA and Infiniband.

In this example, MAC RX block accepts Ethernet frames from the externalEthernet media PHY. MAC RX block performs framing and Ethernet CRCoperations on the raw packets. Filter block discards packets that do notmatch filter rules. Small receive (RX) buffer block provides bufferingto handle message interconnect jitter.

Message mapping block 452 determines an address of a destination messageengine and queue for traffic from the small RX buffer. For example,message mapping block 452 may consider the source address, destinationaddress, and/or payload of the Ethernet frame in identifying adestination message engine and queue for the traffic. The destinationmessage engine and queue may be identified based on an identifier of aVirtual Message Receive Engine (VMRE), queue number (QN), and messagesequence number (MSN). Multiple destination message engines may beavailable, where each destination message engine is associated with acore. Message mapping block 452 may distribute contents of Ethernetframes to cores using a distribution scheme similar to receive sidescaling (RSS) or application targeting routing, although other schemescan be used. Message mapping block 452 may determine contents of messagesegment headers (e.g., non-data portion) using a lookup operation.

Receive (RX) message engine 454 may form message segments and transmitthe messages using an interconnect to the destination message engine(not depicted). The destination message engine (not depicted) mayreceive messages for processing by a core, hardware accelerator, ornetwork protocol offload processor (e.g., iSCSI).

Transmit (TX) message engine 456 may receive messages from a sourcemessage engine. TX message engine 456 may receive message segments fromthe on-die interconnect. TX message engine 456 may examine the payloadof received messages to determine whether the messages contain control,data, or management content. TX message engine 456 directs controlsegments to the control block, management segments to the managementblock, and data segments to the data block.

Messages can be used to transmit control information in lieu of usingmemory based interaction (e.g., PCI). For example, control informationin messages can be used to configure and direct runtime behavior. Forexample, a control message may set link speed.

The control block implements I/O Bridge control functionality, such ascontrolling the other functional blocks and the external Ethernet mediaPHY. The data block may form Ethernet data frames from message segments.Management block may form Ethernet control frames, e.g. PAUSE or otherframes from message segments. The control block may exchange controlmessages with cores or devices elsewhere in the platform. For example,the control block may support control messages that allow configurationof the other blocks in the I/O Bridge.

Small transmit (TX) Buffer block provides buffering to handle messageinterconnect jitter. MAC transmit (TX) block performs framing and CRCoperations before transmitting the packet to an Ethernet media PHY.

In some embodiments, system 450 provides less dedicated silicon in theplatform than a traditional discrete network interface. In someembodiment, system 450 provides a lean path between an Ethernet wire anda core and eliminates latency compared to a traditional networkinterface performing direct memory accesses (DMA). In some embodiments,software can implement OSI Layer 2 features instead of fixed silicongates in a traditional network interface. In some embodiments, unlike atraditional network interface with DMA, system 450 does not requireshared coherent memory with the cores that process the packets.

FIG. 5 depicts a high level block diagram of a message engine inaccordance with an embodiment. Message engine 500 provides transmissionof message segments at the request of a processor, core, or hardwarethread as well as receipt of message segments for processing by aprocessor, core, or hardware thread.

For message transmission, message segmentation block 504 may segmentmessages identified using a send queue 512 for transmission through aninterconnect. Although not depicted, message segmentation block 504 mayuse a data mover to transfer message segments identified using a sendqueue 512 to a location identified using a receive queue (not depicted).Accordingly, memory to memory transfers are made using contents of onelocal SQ and one local RQ and message engines may not interact directlywith the queues of another message engine.

For message segments received from the interconnect, message reassemblyblock 506 may reassemble messages and store the messages into receivequeue 510. Although not depicted, message reassembly block 506 may use adata mover to transfer message segments from a send queue associatedwith another message engine (both not depicted) into a locationidentified using receive queue 510.

Read/write block 502 permits reading of messages for transfer from alocation identified using send queue 512 in virtual memory usingvirtual-to-physical address translation provided from TranslationLookaside Buffer (TLB) 514. Read/write block 502 permits writing ofmessages to a location identified using receive queue 510 in virtualmemory using virtual-to-physical address translation provided fromTranslation Lookaside Buffer (TLB) 514. In one embodiment, read/writeblock 502 is capable of writing message segments to coherent memory andreading message segments from coherent memory in the same manner ashardware threads. Read/write block 502 may share a cache hierarchy andmemory management unit (MMU) with hardware threads in a core.

Translation Lookaside Buffer (TLB) 514 and page miss handler (PMH) 516provide memory management unit (MMU) capabilities. In response tovirtual addresses provided by read/write block 502, TLB 514 convertsvirtual addresses to physical addresses. Hyperthread HT0 or HT1 mayperform address translation of entries in PMH 516. If no entry isavailable in TLB 514, PMH 516 retrieves the addresses from a page tableaddress and stores the addresses in TLB 514. The PMH 516 may be sharedwith hardware threads associated with a core. The PMH 516 may also bededicated for use by one or more message engines.

Interface 508 may provide a physical layer interface between theinterconnect and message segmentation block 504 and message reassemblyblock 506.

FIG. 6 depicts an example format for a context to define an availablevirtual message receive engine (VMRE) in accordance with an embodiment.Field Version (bits 7-0) indicates a version of the protocol. Field ASIDspecifies the Application Space Identifier used by the translationlookaside buffer (TLB) for virtual to physical address translations.Field CR3 specifies the page table address used by a page miss handlerfor virtual to physical address translations. Field Queue Table Addressspecifies the pageable memory address of the array of queues supportedby this VMRE.

A Physical Message Engine (PME) provides virtualized interfaces tohardware threads using Virtual Message Engines (VMEs) at least to sendand receive messages. The PME also provides a physical interface tohardware threads for the purpose of control and configuration of messageengine hardware. The physical interface to the PME may not performvirtual memory translation. Rather, the physical interface of the PMEmay interact with pinned memory in host physical address space. Thephysical interface of the PME may also interact with registers in acore.

A Virtual Message Engine (VME) may perform memory translations using MMUpage tables and interact with virtual memory. As with a software thread,a VME accesses memory in association with an ASID and CR3 pointer to thetop of a page table structure. The VME may share ASID and CR3 valueswith the software threads associated with an address space, though thisis not required. System software may treat the VME as a thread withinthe address space identified by the ASID. For example, system softwaremay keep the page tables utilized by the VME in a consistent state solong as a VMTE or VMRE may send or receive a message. Standard TLBshoot-down rules may apply as with software threads.

FIG. 7 depicts an example format for a no-operation command in a sendqueue in accordance with an embodiment. A send queue (SQ) entry directsthe VMTE to perform a null operation that does not send a message. FieldCommand (bits 7-0) is 0 for a No-op command. Field Immediate Interrupt(II), when set, directs the VME to generate a CQ interrupt immediately,regardless of the interrupt moderation interval.

FIG. 8 depicts an example format for a command in a send queue inaccordance with an embodiment. This command commands a VMTE to transmita message. Field Command (bits 7-0) is 1 for a Send command. FieldImmediate Interrupt (II), when set, directs the VME to generate a CQinterrupt immediately, regardless of the interrupt moderation interval.Field VME Address specifies the destination VMRE for the message. FieldQueue Number specifies the destination queue number for this message.Field size specifies the number of bytes in contiguous virtual memory totransmit. Field address specifies the virtual address in cacheablepageable memory of the data to transmit.

FIG. 9 depicts an example entry in a receive queue in accordance with anembodiment. Field Command (bits 7-0) is 0 for a Receive command. FieldImmediate Interrupt (II), when set, directs the VME to generate a CQinterrupt immediately, regardless of the interrupt moderation interval.Field Size specifies the number of bytes in contiguous virtual memory ofthe receive buffer. Upon completion of the message receive operation forthis descriptor, the VMRE updates the Size field to contain the actualnumber of message segments received. Field Address specifies the virtualaddress in cacheable pageable memory of the receive buffer.

FIG. 10 depicts an example message segment format in accordance with anembodiment. Field RT specifies the Reassembly Tag returned in the CTSMessages. Field Destination VME Address specifies the destination VMREfor the segment. Field MSO specifies the Message Segment Offset of thesegment. The MSO contains the position of this segment relative to thestart of the message. The first segment of the entire message isnumbered 0. The data portion of the segment contains the data payload ofthis segment. The length of the payload is Link Layer specific. Althoughnot depicted, the message segment format may also include a last fieldto indicate that a segment is the last segment of a message.

FIG. 11 depicts an example request to send (RTS) message format inaccordance with an embodiment. Field Command is set to 1 for the RTSMessage. Field Destination VME Address specifies the destination VMREfor the message. Field Source VME Address specifies the source VMTE ofthe message. Field QN specifies the destination Queue Number within thespecified Address Space Identifier. Field MSN specifies the MessageSequence Number within the specified Queue Number. Field RID specifiesthe Request ID that the recipient may return verbatim in the CTS Messageassociated with this request.

A maximum message size may be smaller than the jumbo frame size. In anembodiment, a single RTS message can be transmitted for several adjacentmessages which belong to the same connection, where the adjacentmessages are used to transmit a jumbo frame. A field for “Number ofMessage” could be added in the RTS message to identify the messagenumber associated with a single RTS message and a jumbo frame. Forexample, part of the current Reserved field in FIG. 11 can include theNumber of Message field. In addition, in FIG. 10, a small portion of theMSO field could be used to identify the Number of Message within one RTSor RT.

FIG. 12 depicts an example clear to send (CTS) message format inaccordance with an embodiment. Command field may be set to 2 for the CTSMessage. Field Destination VME Address specifies the destination VMREfor the message. Field Source VME Address specifies the source VMTE ofthe message. Field RT specifies the Reassembly Tag. For subsequentmessage segments, the VMTE places this value in the RT field of everysegment of the message. The upper byte of the RT value is set to 0. RIDfield specifies the Request ID supplied by the VMTE in the RTS message.Field RID allows the VMTE to correlate CTS messages with outstanding RTSmessages.

FIG. 13A depicts an example flow diagram of a process 1300 to transfermessages from an IO device using a message engine in accordance with anembodiment. Block 1302 may include receiving a network protocol unitfrom a network. For example, block 1302 may include receiving anEthernet frame.

Block 1304 may include identifying a virtual message receive engine(VMRE) and destination queue associated with the received frame. TheVMRE may be associated with a message engine that is to receive messagesfor processing by a core. The VMRE and destination queue (QN) may beidentified as described with regard to FIG. 2.

Block 1306 may include segmenting a message for transmission to theVMRE. A virtual message transmit engine (VMTE) may segment the message.A format for message segments may be as described with regard to FIG.10.

Block 1308 may include transmitting each segment to a destinationlocation. The destination location in virtual memory may be identifiedby logic that is to process each received segment based in part ondescriptors in a receive queue, the Reassembly Tag (RT) and MessageSegment Offset (MSO) from the segments.

FIG. 13B depicts an example flow diagrams of a process 1350 to receivemessages using a message engine at an IO device in accordance with anembodiment.

Block 1352 may include reassembling received message segments intocomplete messages. Messages may be received out of order. A MessageSegment Offset (MSO) field in the message may be used to properly orderthe segments into a message.

Block 1354 may include determining the type of message content. Forexample, content can be control, data, or management.

Block 1356 may include preparing a network protocol unit with data orcontrol content for transmission. For example, block 1356 may includepreparing an Ethernet frame with data or control content fortransmission.

Embodiments of the present invention may be provided, for example, as acomputer program product which may include one or more machine-readablemedia having stored thereon machine-executable instructions that, whenexecuted by one or more machines such as a computer, network ofcomputers, or other electronic devices, may result in the one or moremachines carrying out operations in accordance with embodiments of thepresent invention. A machine-readable medium may include, but is notlimited to, floppy diskettes, optical disks, CD-ROMs (Compact Disc-ReadOnly Memories), and magneto-optical disks, ROMs (Read Only Memories),RAMs (Random Access Memories), EPROMs (Erasable Programmable Read OnlyMemories), EEPROMs (Electrically Erasable Programmable Read OnlyMemories), magnetic or optical cards, flash memory, or other type ofmedia/machine-readable medium suitable for storing machine-executableinstructions.

The drawings and the forgoing description gave examples of the presentinvention. Although depicted as a number of disparate functional items,those skilled in the art will appreciate that one or more of suchelements may well be combined into single functional elements.Alternatively, certain elements may be split into multiple functionalelements. Elements from one embodiment may be added to anotherembodiment. For example, orders of processes described herein may bechanged and are not limited to the manner described herein. Moreover,the actions of any flow diagram need not be implemented in the ordershown; nor do all of the acts necessarily need to be performed. Also,those acts that are not dependent on other acts may be performed inparallel with the other acts. The scope of the present invention,however, is by no means limited by these specific examples. Numerousvariations, whether explicitly given in the specification or not, suchas differences in structure, dimension, and use of material, arepossible. The scope of the invention is at least as broad as given bythe following claims.

What is claimed is:
 1. A computer-implemented method comprising:associating a virtual message transmit engine with a first host physicalmessage engine; associating a virtual message receive engine with thevirtual message transmit engine, the virtual message receive engineassociated with a second host physical message engine; transformingcontents of a received network protocol unit into one or more messagesegments; and the virtual message transmit engine requestingtransmission of the one or more message segments using the first hostphysical message engine to a memory region associated with the virtualmessage receive engine, wherein the transmission of the one or moremessage segments to the memory region comprises identification of anaddress of the virtual message receive engine and without the messagesegments identifying the destination memory address of the memoryregion.
 2. The method of claim 1, wherein at least one message segmentcomprises: a link layer field; a destination virtual message engineaddress; a reassembly tag field; a message segment offset field; and adata portion.
 3. The method of claim 2, wherein a location of the memoryregion to store at least one message segment is based in part on thereassembly tag field and the message segment offset field.
 4. The methodof claim 1, further comprising: receiving message segments;re-assembling message segments into a message; forming a second networkprotocol unit based on contents of the message; and causing transmissionof the second network protocol unit.
 5. The method of claim 4, whereincontents of the received message segments comprise at least one ofcontrol, data, and management information.
 6. The method of claim 1,wherein the virtual message transmit engine is associated with a coreand the virtual message receive engine is associated with another core.7. The method of claim 1, further comprising: selecting the virtualmessage receive engine using receive side scaling.
 8. The method ofclaim 1, wherein the first and second host physical message engines donot share the same physical memory and do not maintain memory coherency.9. The method of claim 1, wherein transmission of the one or moremessage segments using the first host physical message engine to amemory region associated with the virtual message receive enginecomprises use of a data mover.
 10. An apparatus comprising: a processorcomprising at least one core; circuitry to receive a network protocolunit and to perform media access control, filtering, and requestbuffering of the network protocol unit; and a physical message engine,wherein one of the at least one core is to associate a virtual messagetransmit engine with the physical message engine, one of the at leastone core is to associate a virtual message receive engine with thevirtual message transmit engine, and the virtual message transmit engineis to request the physical message engine to transmit content of thenetwork protocol unit using at least one message segment to a memoryregion associated with the virtual message receive engine, wherein thetransmission of the at least one message segment to the memory regioncomprises identification of an address of the virtual message receiveengine and without the at least one message segment identifying a memoryaddress of the memory region.
 11. The apparatus of claim 10, wherein atleast one message segment comprises: a link layer field; a destinationvirtual message engine address; a reassembly tag field; a messagesegment offset field; and a data portion.
 12. The apparatus of claim 11,wherein a location of the memory region to store at least one messagesegment is based in part on the reassembly tag field and the messagesegment offset field.
 13. The apparatus of claim 10, wherein the virtualreceive message engine is associated with a core and the virtualtransmit message engine is associated with another core.
 14. Theapparatus of claim 10, wherein a second physical message engine isassociated with the virtual message receive engine and the physicalmessage engine and the second physical message engine do not share thesame physical memory and do not maintain memory coherency.
 15. Theapparatus of claim 10, wherein the transmission of the at least onemessage segment to the memory region comprises use of a data mover. 16.A non-transitory computer-readable medium comprising instructionsthereon, which when executed by a computer, cause the computer to:associate a first message engine with a virtual message transmit engine,associate the virtual message transmit engine with a virtual messagereceive engine; associate a second message engine with the virtualmessage receive engine; request to provide content of a network protocolunit using at least one message segment; and cause the virtual messagetransmit engine to request transmission by the first message engine ofthe at least one message segment to a memory region associated with thevirtual receive message engine, wherein the transmission of the at leastone message segment to the memory region comprises identification of anaddress of the virtual message receive engine and without the at leastone message segment identifying the destination memory address of thememory region.
 17. The non-transitory computer-readable medium of claim16, wherein at least one message segment comprises: a link layer field;a destination virtual message engine address; a reassembly tag field; amessage segment offset field; and a data portion.
 18. The non-transitorycomputer-readable medium of claim 17, wherein a location of the memoryregion to store at least one message segment is based in part on thereassembly tag field and the message segment offset field.
 19. Thenon-transitory computer-readable medium of claim 16, wherein to requesttransmission of the at least one message segment to a memory region, thecomputer is to request use of a data mover to copy content to the memoryregion.
 20. The non-transitory computer-readable medium of claim 16,wherein the first message engine comprises a physical message engine andthe second message engine comprises another physical message engine.